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POSITIONING AND MANIPULATING DETAIL-IN-CONTEXT LENSES IN 2D AND 
3D DATA THROUGH THE APPLICATION OF EYE TRACKING OR POSITION 

TRACKING 

BACKGROUND OF THE INVENTION 

5 Display screens are the primary visual display interface to a computer. One problem with these 
visual display screens is that they are limited in size, thus presenting a challenge to user interface 
design, particularly when larger amounts of information is to be displayed. This problem is 
normally referred to as the "screen real estate problem". 

Well-known solutions to this problem include panning, zooming, scrolling or combinations 
10 thereof. While these solutions are suitable for a large number of visual display applications, 
these solutions become less effective where the visual information is spatially related, such as 
maps, newspapers and such like. In this type of information display, panning, zooming and/or 
scrolling is not as effective as much of the context of the panned, zoomed or scrolled display is 
hidden. 

15 A recent solution to this problem is the application of "detail-in-context" presentation 
techniques. Detail-in-context is the magnification of a particular region of interest (the "focal 
region") in a data presentation while preserving v isibility of the surrounding information. This 
technique has applicability to the display of large surface area media, such as maps, on limited 
size computer screens such as personal digital assistance (PDA's) and cell phones. 

20 In the detail-in-context discourse, differentiation is often made between the terms 
"representation" and "presentation". A representation is a formal system, or mapping, for 
specifying raw information or data that is stored in a computer or data processing system. For 
example, a digital map of a city is a representation of raw data including street names and the 
relative geographic location of streets and utilities. Such a representation may be displayed 

25 visually on computer screen or printed on paper. On the other hand, a presentation is a spatial 
organization of a given representation that is appropriate for the task at hand. Thus, a 
presentation of a representation organizes such things as the point of view and the relative 
emphasis of different parts or regions of the representation. For example, a digital map of a city 
may be presented with a region magnified to reveal street names. 
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In general, a detail-in-context presentation may be considered as a distorted view (or distortion) 
of a portion of the original representation where the distortion is the result of the application of a 
''lens 55 like distortion function to the original representation. A detailed review of various detail- 
in-context presentation techniques such as Elastic Presentation Space ("EPS") may be found in a 
5 publication by Marianne S. T. Carpendale, entitled "A Framework for Elastic Presentation 
Space" (Burnaby, British Columbia: Simon Fraser University, 1999), and incorporated herein by 
reference. 

Development of increasingly powerful computing devices has lead to new possibilities for 
applications of detail-in-context viewing. At the same time, the demand for user control over the 
10 parameters of a detail-in-context lens has increased. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention may best be understood by referring to the following description and 
accompanying drawings which illustrate the invention. In the drawings: 

FIG. 1 is a graphical construction illustrating a 3D perspective viewing frustum in accordance 
1 5 with known elastic presentation space graphics technology; 

FIG. 2 is a cross-sectional view illustrating a presentation in accordance with known elastic 
presentation space graphics technology; 

FIG. 3 is a block diagram illustrating an exemplary data processing system for implementing an 
embodiment of the invention; and, 

20 FIG. 4 a partial screen capture illustrating a GUI having lens control elements for user interaction 
with detail-in-context data presentations in accordance with an embodiment of the invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In the following description, numerous specific details are set forth to provide a thorough 
understanding of the invention. However, it is understood that the invention may be practiced 
25 without these specific details. In other instances, well-known software, circuits, structures and 
techniques have not been described or shown in detail in order not to obscure the invention. The 
term "data processing system" is used herein to refer to any machine for processing data, 



i 



CA 02426512 2003-04-22 



-3- 

including the computer systems and network arrangements described herein. The term "Elastic 
Presentation Space" or "EPS" is used herein to refer to techniques that allow for the adjustment 
of a visual presentation without interfering with the information content of the representation. 
The adjective "elastic" is included in the term as it implies the capability of stretching and 
5 deformation and subsequent return to an original shape. EPS graphics technology is described by 
Carpendale in A Framework for Elastic Presentation Space (Carpendale, Marianne S, T., A 
Framework for Elastic Presentation Space (Bumaby, British Columbia: Simon Fraser 
University, 1999)) which is incorporated herein by reference. In EPS graphics technology, a two- 
dimensional visual representation is placed onto a surface; this surface is placed in three- 
10 dimensional space; the surface, containing the representation, is viewed through perspective 
projection; and the surface is manipulated to effect the reorganization of image details. The 
presentation transformation is separated into two steps: surface manipulation or distortion and 
perspective projection. In the drawings, like numerals refer to like structures or processes. 

Referring to FIG. 1, there is shown a graphical representation 100 of the geometry for 
15 constructing a three-dimensional (3D) perspective viewing frustum 220 relative to an x, y, z 
coordinate system in accordance with known clastic presentation space ("EPS") graphics 
technology. In the EPS, detaii-in-context views of 2D visual representations are created with 
sight-line aligned distortions of a two-dimensional (2D) information presentation surface within 
a 3D perspective viewing frustum 220. In EPS, magnification of regions of interest and the 
20 accompanying compression of the contextual region to accommodate this change in scale are 
produced by the movement of regions of the surface towards the viewpoint 240 located at the 
apex of the pyramidal shape 220 containing the frustum. The process of projecting these 
transformed layouts via a perspective projection results in a new 2D layout which includes the 
zoomed and compressed regions. The use of the third dimension and perspective distortion to 
25 provide magnification in EPS provides a meaningful metaphor for the process of distorting the 
infonnation presentation surface. The 3D manipulation of the information presentation surface in 
such a system is an intermediate step in the process of creating a new 2D layout of the 
information. 

Referring to FIG. 2, there is shown geometrical representation of a presentation 200 in 
30 accordance with known EPS graphics technology. EPS graphics technology employs viewer- 
aligned perspective projections to produce detail-in-context presentations in a reference view 
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plane 201 which may be viewed on a display. Undistorted 2D data points are located in a basal 
plane 210 of a 3D perspective viewing volume or frustum 220 which is defined by extreme rays 
221 and 222 and the basal plane 210. A viewpoint ("VP") 240 is located above the centre point 
of the basal plane 210 and reference view plane 201. Points in the basal plane 210 are displaced 
5 upward onto a distorted surface 230 which is defined by a general 3D distortion function (i.e. a 
detail-in-ccntext distortion basis function). The direction of the viewer-aligned perspective 
projection corresponding to the distorted surface 230 is indicated by the line FPo - FP 231 drawn 
from a point FPo 232 in the basal plane 210 through the point FP 233 which corresponds to the 
focus or focal region or focal point of the distorted surface 23(5. 

10 To reiterate, EPS refers tc a collection of know-how and techniques for performing "detail-in- 
context viewing" (also known as "multi-scale viewing" and "distortion viewing") of information 
such as images, maps, and text, using a projection technique summarized below. EPS is 
applicable to multidimensional data and is well suited to implementation on a computer for 
dynamic detail-in-context display on an electronic display surface such as a monitor. In the case 

15 of two dimensional data, EPS is typically characterized by magnification of areas of an image 
where detail is desired, in combination with compression of a restricted range of areas of the 
remaining information (the "context"), the end result typically giving the appearance of a lens 
having been applied to the display surface. EPS has numerous advantages over conventional 
zoom, pan, and scroll technologies, including the capability of preserving the visibility of 

20 information outside the local region of interest. 

In general, in EPS, the source image to be viewed is located in the basal plane. Magnification 
and compression are achieved through elevating elements of the source image relative to the 
basal plane, and then projecting the resultant distorted surface onto the reference view plane. 
EPS performs detaii-in-context presentation of n-diniensional data through the use of a procedure 
25 wherein the data is mapped into a region in an (n+1) dimensional space, manipulated through 
perspective projections in the (n-f 1) dimensional space, and then finally transformed back into n- 
dimensional space for presentation. 

For example, and referring to FIGS. 1 and 2. in two dimensions, EPS can be implemented 
through the projection of an image onto a reference plane 201 in the following manner. The 
30 source image is located on a basal plane 210, and those regions of interest 233 of the image for 
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which magnification is desired are elevated so as to move them closer to a reference plane 
situated between the reference viewpoint 240 and the reference view plane (RVP) 201. 
Magnification of the "focal region" 233 closest tc the RVP varies inversely with distance from 
the RVP 201. As shown in FIGS. I and 2, compression of regions outside the focal region 233 is 
5 a function of both distance from the RVP 201, and the gradient of the function describing the 
vertical distance from the RVP 201 with respect to horizontal distance from the focal region 233. 
The resultant combination of magnification and compression of the image as seen from the 
reference viewpoint 240 results in a lens-like effect similar to that of a magnifying glass applied 
to the image, and the resultant distorted image may be referred to as a "pliable display surface". 
10 Hence, the various functions used to vary the magnification and compression of the image via 
vertical displacement from the basal plane 210 arc described as lenses, lens types, or lens 
functions. Lens functions that describe basic lens types with point and circular focal regions, as 
well as certain more complex lenses and advanced capabilities such as folding, have previously 
been described by Carpendale. 

15 System. Referring to FIG. 3, there is shown a block diagram of an exemplary data processing 
system 300 for implementing an embodiment of the invention. The data processing system is 
suitable for implementing EPS technology in conjunction with eye tracking, position tracking, 
and a graphical user interface ("GUI"). The data processing system 300 includes an input device 
310, a central processing unit or CPU 320, memory 330, a display 340, and eye tracking and/or 

20 position tracking hardware 351, 352. The input device 310 may include a keyboard, mouse, 
trackball, or similar device. The CPU 320 may include dedicated coprocessors and memory 
devices. The memory 330 may include RAM, ROM, databases, or disk devices. The display 340 
may include a computer screen or terminal device. And, the eye tracking and'or position tracking 
hardware 351, 352 may include cameras, touch-screens, wands, and electromagnetic sensors 

25 with appropriate controllers. The data processing system 300 has stored therein data representing 
sequences of instructions which when executed cause the method described herein to be 
performed. Of course, the data processing system 300 may contain additional software and 
hardware a description of which is not necessary for understanding the invention. 

GUI with Lens Control Elements. As mentioned, detail-in-context presentations of data using 
30 techniques such as pliable surfaces, as described by Carpendale, are useful in presenting large 
amounts of information on limited-size display surfaces. Detail-in-context views allow 
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magnification of a particular region of interest (the "focal region") 233 in a data presentation 
while preserving visibility of the surrounding information 210. In the following, a GUI having 
lens control elements that can be implemented in software and applied to the control of detail-in- 
context data presentations, including EPS and pliable surfaces, is described. The software can be 
5 loaded into and run by the exemplary data processing system 300 of FIG. 3. 

Referring to FIG. 4, there is a partial screen capture illustrating a GUI 400 having lens control 
elements for user interaction with detail-in-context data presentations in accordance with an 
embodiment of the invention. Detail-in-context data presentations are characterized by 
magnification of areas of an image where detail is desired, in combination with compression of a 

10 restricted range of areas of the remaining information (i.e. the "context"), the end result typically 
giving the appearance of a 'lens" having been applied to the display screen surface. This "lens" 
410 includes a "focal region" 420 having high magnification, a surrounding "shoulder region" 
430 where information is typically visibly compressed, and a "base" 412 surrounding the 
shoulder region 430 and defining the extent of the lens 410. In FIG. 4, the lens 410 is shown with 

15 a circular shaped base 412 (or outline) and with a focal region 420 lying near the center of the 
lens 418. However, the lens 410 and focal region 420 may have any desired shape. Referring 
again to FIG. 2, the iens 410 corresponds to the projection of the distorted surface 230 and focal 
region 233 onto the reference plane 201. 

In general, the GUI 400 has lens control elements that, in combination, provide for the 
20 interactive control of the lens 410. The effective control of the characteristics of the lens 410 by a 
user (i.e. dynamic interaction with a detail-in-context lens) is advantageous. At any given time, 
one or more of these lens control elements may be made visible to the user on the display surface 
340 by appearing as overlay icons on the lens 410. Interaction with each element is performed 
via the motion of a pointing device 310 (e.g. mouse), with the motion resulting in an appropriate 
25 change in the corresponding lens characteristic. As will be described, selection of which lens 
control element is actively controlled by the motion of the pointing device 310 at any given time 
is determined by the proximity of the icon representing the pointing device 310 on the display 
surface 340 (e.g. cursor) to the appropriate component of the lens 410. For example, "dragging" 
of the pointing device at the periphery of the bounding rectangle of the lens base 412 causes a 
30 corresponding change in the size of the lens 410 (i.e. "resizing"). 'ITius, the GUI 400 provides the 
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user with a visual representation of which lens control element is being adjusted through the 
display of one or more corresponding icons. 

For ease of understanding, the following discussion will he in the context of using a two- 
dimensional pointing device 310 that is a mouse, but it will be understood that the invention may 
5 be practiced with other 2-D or 3-D (or even greater numbers of dimensions) pointing devices 
including a trackball and keyboard. 

A mouse 310 controls the position of a cursor icon 401 that is displayed on the display screen 
340. The cursor 401 is moved by moving the mouse 310 over a flat surface, such as the top of a 
desk, in the desired direction of movement of the cursor 401. Thus, the two-dimensional 
10 movement of the mouse 310 on the flat surface translates into a corresponding two-dimensional 
movement of the cursor 405 on the display screen 340. 

A mouse 310 typically has one or more finger actuated control buttons (i.e. mouse buttons). 
While the mouse buttons can be used for different functions such as selecting a menu option 
pointed at by the cursor 401, the disclosed invention may use a single mouse button to "select" a 

1 5 lens 410 and to trace the movement of the cursor 401 along a desired path. Specifically, to select 
a lens 410, the cursor 401 is first located within the extent of the lens 410. In other words, the 
cursor 401 is "pointed" at the lens 410. Next, the mouse button is depressed and released. That 
is. the mouse button is "clicked". Selection is thus a point and click operation. To trace the 
movement of the cursor 401, the cursor 401 is located at the desired starting location, the mouse 

20 button is depressed to signal the computer 320 to activate a lens control element, and the mouse 
310 is moved while maintaining the button depressed. After the desired path has been traced, the 
mouse button is released. This procedure is often referred to as "clicking" and "dragging" (i.e. a 
click and drag operation). It will be understood that a predetermined key on a keyboard 310 
could also be used to activate a mouse click or drag. In the following, the term "clicking" will 

25 refer to the depression of a mouse button indicating a selection by the user and the term 
"dragging" will refer to the subsequent motion of the mouse 310 and cursor 401 without the 
release of the mouse button. 

The GUI 400 may include the following lens control elements: move, pickup, resize base, resize 
focus, fold, and magnify. Each of these lens control elements has at least one lens control icon or 
30 alternate cursor icon associated with it. In general, when a lens 410 is selected by a user through 
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a point and click operation, the following lens control icons may be displayed over the lens 410: 
pickup icon 450, base outline icon 412, base bounding rectangle icon 411, focal region bounding 
rectangle icon 421, handle icons 481, 482, 491, and magnify slide bar icon 440. Typically, these 
icons are displayed simultaneously after selection of the lens 410, In addition, when the cursor 
5 401 is located within the extent of a selected lens 410, an alternate cursor icon 460, 470, 480, 490 
may be displayed over the lens 410 to replace the cursor 401 or may be displayed in combination 
with the cursor 401. These lens control elements, corresponding icons, and their effects on the 
characteristics of a iens 410 are described below with reference to FIG. 4. 

In general, when a lens 410 is selected by a point and click operation, bounding rectangle icons 
10 411, 421 are displayed surrounding the base 412 and focal region 420 of the selected lens 410 to 
indicate that the lens 410 has been selected. With respect to the bounding rectangles 411, 421 
one might view them as glass windows enclosing the lens base 412 and focal region 420, 
respectively. The bounding rectangles 411, 421 include handle icons 481, 482, 491 allowing for 
direct manipulation of the enclosed base 412 and foca! region 420 as will be explained below. 
15 Thus, the bounding rectangles 411, 421 not only inform the user that the lens 410 has been 
selected, but also provide the user with indications as to what manipulation operations might be 
possible for the selected lens 410 though use of the displayed handles 481, 482, 491. Note that it 
is well within the scope of the present invention to provide a bounding region having a shape 
other than generally rectangular. Such a bounding region could be of any of a great number of 
20 shapes including oblong, oval, ovoid, conical, cubic, cylindrical, polyhedral, spherical, etc. 

Moreover, the cursor 401 provides a visual cue indicating the nature of an available lens control 
element. As such, the cursor 401 will generally change in form by simply pointing to a different 
lens control icon 450, 412, 411, 421, 481, 482, 491, 440. For example, when resizing the base 
412 of a lens 410 using a corner handle 491, the cursor 401 will change form to a resize icon 490 
25 once it is pointed at (i.e. positioned over) the corner handle 491. The cursor 401 will remain in 
the form of the resize icon 490 until the cursor 401 has been moved away from the corner handle 
491. 

Move. Lateral movement of a lens 410 is provided by the move lens control element of the GUI. 
Tnis functionality is accomplished by the user firsr selecting the lens 410 through a point and 
30 click operation. Then, the user points to a point within the lens 410 that is other than a point 
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lying on a lens control icon 450, 412, 411, 421, 481, 482, 491, 440. When the cursor 401 is so 
located, a move icon 460 is displayed over the lens 419 to replace the cursor 401 or may be 
displayed in combination with the cursor 401. The move icon 460 not only informs the user that 
the lens 410 may be moved, but also provides the user with indications as to what movement 
5 operations are possible for the selected lens 410. For example, the move icon 460 may include 
arrowheads indicating up, down, left, and right motion. Next, the lens 410 is moved by a click 
and drag operation in which the user clicks and drags the lens 410 to the desired position on the 
screen 340 and then releases the mouse button 310. The lens 410 is locked in its new position 
until a further pickup and move operation is performed. 

Pickup. Lateral movement of a lens 410 is also provided by the pickup lens control element of 
the GUI. This functionality is accomplished by the user first selecting the lens 410 through a 
point and click operation. As mentioned above, when the iens 410 is selected a pickup icon 450 
is displayed over the lens 410 near the centre of the lens 410. Typically, the pickup icon 450 will 
be a crosshairs. In addition, a base outline 412 is displayed over the lens 410 representing the 
base 412 of the lens 410. The crosshairs 450 and lens outline 412 not only inform the user that 
the lens has been selected, but also provides the user with an indication as to the pickup 
operation that is possible for the selected lens 410. Next, the user points at the crosshairs 450 
with the cursor 401. Then, the lens outline 412 is moved by a click and drag operation in which 
the user clicks and drags the crosshairs 450 to the desired position on the screen 340 and then 
releases the mouse button 310. The full lens 410 is then moved to the new position and is locked 
there until a further pickup operation is performed. In contrast to the move operation described 
above, with the pickup operation, it is the outline 412 of the lens 410 that the user repositions 
rather than the full lens 410. 

Resize Base. Resizing of the base 412 (or outline) of a lens 410 is provided by the resize base 
25 lens control element of the GUI. After the lens 410 is selected, a bounding rectangle icon 411 is 
displayed surrounding the base 412. The bounding rectangle 411 includes handles 491. These 
handles 491 can be used to stretch the base 412 taller or shorter, wider or narrower, or 
proportionally larger or smaller. The comer handles 491 will keep the proportions the same 
while changing the size. The middle handles (not shown) will make the base 412 taller or shorter, 
30 wider or narrower. Resizing the base 412 by the corner handles 491 will keep the base 412 in 
proportion. Resizing the base 412 by the middle handles (not shown) will change the proportions 
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of the base 412. That is, the middle handles (not shown) change the aspect ratio of the base 412 
(i.e. the ratio between the height and the width of the bounding rectangle 411 of the base 412). 
When a user points at a handle 49i with the cursor 401 a resize icon 490 may be displayed over 
the handle 491 to replace the cursor 401 or may be displayed in combination with the cursor 401. 
5 The resize icon 490 not only informs the user that the handle 491 may be selected, but also 
provides the user with indications as to the resizing operations that are possible with the selected 
handle. For example, the resize icon 490 tor a comer handle 491 may include arrows indicating 
proportional resizing. The resize icon (not shown) for a middle handle (not shown) may include 
arrows indicating width resizing or height resizing. After pointing at the desired handle 491, the 
10 user would click and drag the handle 491 until the desired shape and size for the base 412 is 
reached. Once the desired shape and size are reached, the user would release the mouse button 
310. The base 412 of the lens 410 is then locked m its new size and shape until a further base 
resize operation is performed. 

Resize Focus. Resizing of the focai region 420 of a lens 410 is provided by the resize focus lens 

15 control element of the GUI. After the lens 410 is selected, a bounding rectangle icon 421 is 
displayed surrounding the focal region 420. The bounding rectangle 421 includes handles 481, 
482. These handles 481, 482 can be used to stretch the focal region 420 tailer or shorter, wider 
or narrower, or proportionally larger or smaller. The corner handles 481 will keep the 
proportions the same while changing the size. The middle handles 482 will make the focal region 

20 420 taller or shorter, wider or narrower. Resizing the focal region 420 by the corner handles 481 
will keep the focal region 420 in proportion. Resizing the focal region 420 by the middle handles 
482 will change the proportions of the focal region 420. That is. the middle handles 482 change 
the aspect ratio of the focal region 420 (i.e. the ratio between the height and the width of the 
bounding rectangle 421 of the focal region 420), When a user points at a handle 481, 482 with 

25 the cursor 401 a resize icon 480 may be displayed over the handle 481, 482 to replace the cursor 
401 or may be displayed in combination with the cursor 401. The resize icon 480 not only 
informs the user that a handle 481, 482 may be selected, but also provides the user with 
indications as to the resizing operations that are possible with the selected handle. For example, 
the resize icon 480 for a corner handle 481 may include arrows indicating proportional resizing. 

30 The resize icon 480 for a middle handle 482 may include arrows indicating width resizing or 
height resizing. After pointing at the desired handle 481, 482, the user would click and drag the 



! 



CA 02426512 2003-04-22 
-11- 

handle 481, 482 until the desired shape and size for the focal region 420 is reached, Once the 
desired shape and size are reached, the user would release the mouse button 310, The focal 
region 420 is then locked in its new size and shape until a further focus resize operation is 
performed. 

5 Fold. Folding of the focal region 420 of a lens 410 is provided by the fold control element of the 
GUI. In general, control of the degree and direction of folding (i.e. skewing of the viewer aligned 
vector 231 as described by Carpendale) is accomplished by a click and drag operation on a point 
471, other than a handle 481, 482, on the bounding rectangle 421 surrounding the focal region 
420. The direction of folding is determined by the direction in which the point 471 is dragged. 

10 The degree of folding is determined by the magnitude of the translation of the cursor 401 during 
the drag. In general, the direction and degree of folding corresponds to the relative displacement 
of the focus 420 with respect to the lens base 410. In other words, and referring to FIG. 2, the 
direction and degree of folding corresponds to the displacement of the point FP 233 relative to 
the point FPo 232, where the vector joining the points FPo 232 and FP 233 defines the viewer 

15 aligned vector 231. 

In particular, after the lens 410 is selected, a bounding rectangle icon 421 is displayed 
surrounding the focal region 420. The bounding rectangle 42 J includes handles 481, 482. When 
a user points at a point 471, other than a handle 481, 482, on the bounding rectangle 421 
surrounding the focal region 420 with the cursor 401, a fold icon 470 may be displayed over the 

20 point 471 to replace the cursor 401 or may be displayed in combination with the cursor 401. The 
fold icon 470 not only informs the user that a point 471 on the bounding rectangle 421 may be 
selected, but also provides the user with indications as to what foid operations are possible. For 
example, the fold icon 470 may include arrowheads indicating up, down, left, and right motion. 
3y choosing a point 471, other than a handle 481, 482. on the bounding rectangle 421 a user may 

25 control the degree and direction of folding. To control the direction of folding, the user would 
click on the point 471 and drag in the desired direction of folding. To control the degree of 
folding, the user would drag to a greater or lesser degree in the desired direction of folding. Once 
the desired direction and degree of folding is reached, the user would release the mouse button 
310. The lens 410 is then locked with the selected fold until a further fold operation is 

30 performed. 
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Magnify. Magnification of the lens 410 is provided by the magnify lens control element of the 
GUI. After the lens 410 is selected, the magnify control is presented to the user as a slide bar 
icon 440 near or adjacent to the lens 410 and typically to one side of the lens 410. Sliding the bar 
441 of the slide bar 440 results in a proportional change in the magnification of the lens 410. The 

5 slide bar 440 not only informs the user that magnification of the lens 410 may be selected, but 
also provides the user with an indication as to what level of magnification is possible. The slide 
bar 440 includes a bar 441 that may be slid up and down, or left and right, to adjust and indicate 
the level of magnification. To control the level of magnification, the user would click on the bar 
441 of the slide bar 440 and drag in the direction of desired magnification level. Once the desired 

10 level of magnification is reached, the user would release the mouse button 310. The lens 410 is 
then locked with the selected magnification until a further magnification operation is performed. 

Typically, the focal region 420 is an area of the lens 410 having constant magnification (i.e. if 
the focal region is a plane). Again referring to FIGS. 1 and 2, magnification of the focal region 
420, 233 varies inversely with the distance from the focal region 420, 233 to the reference view 
15 plane (RVP) 201. Magnification of areas lying in the shoulder region 430 of the lens 410 also 
varies inversely with their distance from the RVP 201. Thus, magnification of areas lying in the 
shoulder region 430 will range from unity at the base 412 to the level of magnification of the 
focal region 420. 

Icon Hiding. Advantageously, a user may choose to hide one or more lens control icons 450, 
20 412, 411, 421, 481, 482, 491, 440 shown in FIG. 4 from view so as not to impede the user's view 
of the data (i.e. visual information) within the lens 410. This may be helpful, for example, during 
a move operation. A user may select this option through means such as a menu or lens property 
dialog box. 

Positioning and Manipulating Detail-Fn-Context Lenses in 2D And 3D Data Through the 
25 Application of Eye Tracking or Position Tracking. As mentioned, detail-in-context viewing is a 
technique applicable tc 2D or 3D data that allows for data to magnified locally while maintaining 
continuity of data. In two dimensions, an arbitrarily shaped region of interest is magnified in 
place within the data, while a surrounding band of variable magnification connects the region of 
interest with the surrounding image. The magnified region is referred to as the "focal region," the 
30 surrounding band of variable magnification is referred to as the "shoulder region," and the 
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surrounding image is referred to as the "base image." The focal region and shoulder region 
together comprise the lens. When applied to three dimensional data, the lens concept itself can be 
extended into three dimensions. The lens typically exists as a cylinder (although it can be other 
shapes) of magnification that extends from the viewpoint through the point of interest in the data. 
5 The cylinder of magnification can be of finite or infinite depth. The magnification cylinder can 
act on the data by magnifying objects, displacing objects, or deforming individual objects. 

In either two dimensions or three dimensions, it is desirable to have a method of positioning and 
manipulating the lens. Such parameters as lens position, lens size, and lens magnification may 
need to be adjusted by the user. The present invention deals with the manipulation of lens 
10 parameters through the use of eye tracking technology or the sensing of position of physical 
objects. 

Eye Tracking and Detail-In-Ccntext Lerises. Eye tracking technology typically consists of one or 
more cameras that survey a scene, identify the eyes of people in the scene, and through some 
technique determine the direction that the eyes are looking. For desktop applications the camera 
1 5 typically sits on a desk near a monitor and is pointed towards the chair at which a user would sit. 
When the user sits at the chair and uses the computer, the eye tracker can identify the point on 
the screen at which the user is looking. According to one aspect of the present invention, desktop 
eye-tracking is combined with dctail-in-contcxt viewing. Note that the techniques described 
below can be applied to both 2D and 3D lenses. 

20 i . Positioning of Lenses: The eye tracker is able to determine the point on the screen 

at which a user is looking. With a detail-in-contcxt lens on the screen, it is 
possible to position said lens at the location on the screen at which the user is 
looking. 

2. Manipulation of Lenses: Eye tracking could be used to manipulate parameters 
25 other than lens position. For example, the eye gaze point of interest and eye 

movement could be used to change parameters such as lens size and 
magnification level. 



3. 



Eye Gestures: Eye gestures could be used to change mode or perform operations. 
For example, a prolonged blink could be used to activate or deactivate a lens. 
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Movement Damping: As human gaze tends to include a certain amount a noise 
(unintended minor fluctuations in gaze direction), il may be desirable to apply a 
damping function to the gaze direction before manipulating lens parameters. 

Large Screen Applications: Combined use of eye tracking and detail-in-context 
ienses could be applied to large displays, such as wall-mounted displays or digital 
whiteboards. 

Multi-User Scenario: Combined use of eye tracking and dctail-in-context lenses 
could be used by multiple users interacting with one screen. One lens per user 
could be manipulated. 

Remote Gaze Awareness: The position and state of a lens as determined by gaze 
direction could be transmitted over a network to a remote machine in order to 
impart an awareness to the remote user of the gaze direction of the local user. 

The techniques described above for use in desktop and large screen environments can be 
extended for use in immersive virtual reality environments. Instead of the eye tracker 
15 determining the point of interest on the screen through eye tracking, it would determine the 
direction of gaze through a virtual environment. This could be used to position and manipulate 
lenses. Furthermore, the use of an eye tracker that can detect focal depth (through die tracking of 
both eyes of a user) would be able to position a lens at a depth defined by the focal depth. 

Position Tracking and Delail-In-Contexl Lenses. There are a variety of technologies that allow 
20 for the tracking of physical objects in three dimensions. These technologies include camera 
technologies that track objects through optical recognition, and magnetic technologies that track 
sensors through measurement of magnetic fields. According to one aspect of the invention, 
detaii-in-contexi lenses are manipulated through the measurement of position, orientation, and 
movement of physical objects controlled by a user. Examples of embodiments of this invention 
25 are as follows: 

1. A user is examining a 2D map on a screen. Sensors detect the position and 
orientation of the user's finger. When the user points at a location on the screen, a 
lens appears at that point. If the user points at a different location, the lens moves 
to that location. 



5. 
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6. 
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2. A user is examining a 3D model on a screen. Sensors detect the position and 
orientation of a user's finger. When the user points at a location on the screen, a 
3D lens is inserted in the data, originating from tip of the finger, and progressing 
through the data in a direction determined by the finger's orientation. 

5 3. A user is in an immersive virtual reality environment, observing a 3D model 

floating within reach. The user reaches out and touches a finger within the model 
at a region of interest. A 3D lens is inserted in the model at the point touched by 
the user, oriented towards the user's point of view. 

Alternative versions of the three previous examples can be constructed to include the sensing of 
10 objects other than a user's finger. Examples of other objects that could be used to direct a lens 
include wands, remote controls, and laser pens. 

Data Carrier Product. The sequences of instructions which when executed cause the method 
described herein to be performed by the exemplary data processing system of FIG. 3 can be 
contained in a data carrier product according to one embodiment of the invention. This data 
1 5 carrier product can be loaded into and run by the exemplary data processing system of FIG. 3. 

Computer Software Product. The sequences of instructions which when executed cause the 
method described herein to be performed by the exemplary data processing system of FIG. 3 can 
be contained in a computer software product according to one embodiment of the invention. This 
computer software product can be loaded into and run by the exemplary data processing system 
20 of FIG. 3. 

Integrated Circuit Product, The sequences of instructions which when executed cause the 
method described herein to be performed by the exemplary data processing system of FIG. 3 can 
be contained in an integrated circuit product including a coprocessor or memory according to one 
embodiment of the invention. This integraisd circuit product can be installed in the exemplary 
25 data processing system of FIG, 3 . 
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